{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "aXeleRate_test_classifier.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "1rCJbj9BGoDxEt1ERSK3onxShVBv9LS7B",
      "authorship_tag": "ABX9TyNi51OCg86+9D/KZmhK6dS+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AIWintermuteAI/aXeleRate/blob/master/resources/aXeleRate_standford_dog_classifier.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hS9yMrWe02WQ",
        "colab_type": "text"
      },
      "source": [
        "## Standford Dog Breed Classification model Training and Inference\n",
        "\n",
        "In this notebook we will use axelerate Keras-based framework for AI on the edge to quickly setup model training and then after training session is completed convert it to .tflite and .kmodel formats.\n",
        "\n",
        "First, let's take care of some administrative details. \n",
        "\n",
        "1) Before we do anything, make sure you have choosen GPU as Runtime type (in Runtime - > Change Runtime type).\n",
        "\n",
        "2) We need to mount Google Drive for saving our model checkpoints and final converted model(s). Press on Mount Google Drive button in Files tab on your left. \n",
        "\n",
        "In the next cell we clone axelerate Github repository and import it. \n",
        "\n",
        "**It is possible to use pip install or python setup.py install, but in that case you will need to restart the enironment.** Since I'm trying to make the process as streamlined as possibile I'm using sys.path.append for import."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y07yAbYbjV2s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "!git clone https://github.com/AIWintermuteAI/aXeleRate.git\n",
        "import sys\n",
        "sys.path.append('/content/aXeleRate')\n",
        "from axelerate import setup_training,setup_inference"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5TBRMPZ83dRL",
        "colab_type": "text"
      },
      "source": [
        "At this step you typically need to get the dataset. You can use !wget command to download it from somewhere on the Internet or !cp to copy from My Drive as in this example\n",
        "```\n",
        "!cp -r /content/drive/'My Drive'/pascal_20_segmentation.zip .\n",
        "!unzip --qq pascal_20_segmentation.zip\n",
        "```\n",
        "For this notebook we will use Standford Dog Breed Classification dataset for fine-grained classification, which you can download here:\n",
        "http://vision.stanford.edu/aditya86/ImageNetDogs/\n",
        "\n",
        "In the next cell we will download the same dataset, but with training/validation split already done - I shared on my Google Drive. We will also download pre-trained model to demonstrate inference results.\n",
        "\n",
        "Let's visualize our classification validation dataset with visualize_dataset function, which will search for all images in folder and display num_imgs number of images with class overlayer over the image.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_tpsgkGj7d79",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!gdown https://drive.google.com/uc?id=1qq758Tjsfm7Euu9ev7hSyLkMj63YC9ST  #dog breed classification dataset\n",
        "!gdown https://drive.google.com/uc?id=1dFnDCOxws2uX4ZpauSPC6r6jdjHoJw_p  #pre-trained model\n",
        "!unzip --qq dogs_classification.zip\n",
        "\n",
        "from axelerate.networks.classifier.data_gen import visualize_dataset\n",
        "\n",
        "visualize_dataset('dogs_classification/imgs_validation', 10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S1oqdtbr7VLB",
        "colab_type": "text"
      },
      "source": [
        "Next step is defining a config dictionary. Most lines are self-explanatory.\n",
        "\n",
        "Type is model frontend - Classifier, Detector or Segnet\n",
        "\n",
        "Architecture is model backend (feature extractor) \n",
        "\n",
        "- Full Yolo\n",
        "- Tiny Yolo\n",
        "- MobileNet1_0\n",
        "- MobileNet7_5 \n",
        "- MobileNet5_0 \n",
        "- MobileNet2_5 \n",
        "- SqueezeNet\n",
        "- NASNetMobile\n",
        "- DenseNet121\n",
        "- ResNet50\n",
        "\n",
        "**Note that while you can train any network type with any backend (Tiny YOLO + Classifier, NASNETMobile +  Detector, DenseNet121 + Segnet and so on), some converters do not support larger networks! E.g. K210 converter only supports MobileNet and TinyYOLO backends.**\n",
        "\n",
        "Fully_connected is number of neurons in classification layers as list.\n",
        "\n",
        "Dropout value is dropout in classification layers.\n",
        "\n",
        "actual_epoch is number of epochs to train, noramlly good starting value is 50 - 100\n",
        "\n",
        "train_times is a multiplier for training dataset, i.e. how many times to repeat the dataset during one epoch. Useful when you apply augmentations to image. Normally between 1 and 3 is okay. If you have big dataset, can leave at 1.\n",
        "\n",
        "**Since it is an example notebook, we will use pretrained weights and set all layers of the model to be \"frozen\"(non-trainable), except for the last one. Also we set learning rate to very low value, that will allow us to see the perfomance of pretrained model** "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jw4q6_MsegD2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "config = {\n",
        "    \"model\" : {\n",
        "        \"type\":                 \"Classifier\",\n",
        "        \"architecture\":         \"NASNetMobile\",\n",
        "        \"input_size\":           224,\n",
        "        \"fully-connected\":      [],\n",
        "        \"labels\":               [],\n",
        "        \"dropout\" : \t\t0.2\n",
        "    },\n",
        "     \"weights\" : {\n",
        "            \"full\":   \t\t\t\t\"/content/Classifier_best_val_accuracy.h5\",\n",
        "            \"backend\":   \t\t    \"imagenet\",\n",
        "            \"save_bottleneck\":      False\n",
        "        \n",
        "    },\n",
        "    \"train\" : {\n",
        "        \"actual_epoch\":         1,\n",
        "        \"train_image_folder\":   \"dogs_classification/imgs\",\n",
        "        \"train_times\":          1,\n",
        "        \"valid_image_folder\":   \"dogs_classification/imgs_validation\",\n",
        "        \"valid_times\":          1,\n",
        "        \"valid_metric\":         \"val_accuracy\",\n",
        "        \"batch_size\":           16,\n",
        "        \"learning_rate\":        1e-6,\n",
        "        \"saved_folder\":   \t\tF\"/content/drive/My Drive/dogs_classifier\",\n",
        "        \"first_trainable_layer\": \"dense_1\",\n",
        "        \"augumentation\":\t\t\t\tTrue\n",
        "    },\n",
        "    \"converter\" : {\n",
        "        \"type\":   \t\t\t\t[\"tflite\"]\n",
        "    }\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kobC_7gd5mEu",
        "colab_type": "text"
      },
      "source": [
        "Let's check what GPU we have been assigned in this Colab session, if any."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rESho_T70BWq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.python.client import device_lib\n",
        "device_lib.list_local_devices()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cWyKjw-b5_yp",
        "colab_type": "text"
      },
      "source": [
        "Finally we start the training by passing config dictionary we have defined earlier to setup_training function. The function will start the training with Checkpoint, Reduce Learning Rate on Plateu and Early Stopping callbacks. Every time our validation metric(in this config set to \"val_accuracy\") improves, the model is saved with Checkpoint callback. After the training has stopped, it will convert the best model into the format you have specified in config and save it to the project folder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "deYD3cwukHsj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import backend as K \n",
        "K.clear_session()\n",
        "model_path = setup_training(config_dict=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ypTe3GZI619O",
        "colab_type": "text"
      },
      "source": [
        "After training it is good to check the actual perfomance of your model by doing inference on your validation dataset and visualizing results. This is exactly what next block does. Our model used pre-trained weights and since all the layers,except for the last one were set as non-trainable and we set the learning rate to a very low value, we are just observing the perfomance of the model that was trained before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jE7pTYmZN7Pi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import backend as K \n",
        "K.clear_session()\n",
        "setup_inference(config, model_path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fn7H0V4SEOd_",
        "colab_type": "text"
      },
      "source": [
        "To train the model from scratch use the following config and then run the cells with training and (optinally) inference functions again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oT87SwQ6EQB8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "config = {\n",
        "    \"model\" : {\n",
        "        \"type\":                 \"Classifier\",\n",
        "        \"architecture\":         \"NASNetMobile\",\n",
        "        \"input_size\":           224,\n",
        "        \"fully-connected\":      [],\n",
        "        \"labels\":               [],\n",
        "        \"dropout\" : \t\t0.2\n",
        "    },\n",
        "     \"weights\" : {\n",
        "            \"full\":   \t\t\t\t\"\",\n",
        "            \"backend\":   \t\t    \"imagenet\",\n",
        "            \"save_bottleneck\":      False\n",
        "        \n",
        "    },\n",
        "    \"train\" : {\n",
        "        \"actual_epoch\":         50,\n",
        "        \"train_image_folder\":   \"dogs_classification/imgs\",\n",
        "        \"train_times\":          1,\n",
        "        \"valid_image_folder\":   \"dogs_classification/imgs_validation\",\n",
        "        \"valid_times\":          1,\n",
        "        \"valid_metric\":         \"val_accuracy\",\n",
        "        \"batch_size\":           16,\n",
        "        \"learning_rate\":        1e-3,\n",
        "        \"saved_folder\":   \t\tF\"/content/drive/My Drive/dogs_classifier\",\n",
        "        \"first_trainable_layer\": \"\",\n",
        "        \"augumentation\":\t\t\t\tTrue\n",
        "    },\n",
        "    \"converter\" : {\n",
        "        \"type\":   \t\t\t\t[\"tflite\"]\n",
        "    }\n",
        "}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NQjvas2UEe8l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import backend as K \n",
        "K.clear_session()\n",
        "model_path = setup_training(config_dict=config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJJWjuRaEfkj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import backend as K \n",
        "K.clear_session()\n",
        "setup_inference(config, model_path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5YuVe2VD11cd",
        "colab_type": "text"
      },
      "source": [
        "Good luck and happy training! Have a look at these articles, that would allow you to get the most of Google Colab or connect to local runtime if there are no GPUs available;\n",
        "\n",
        "https://medium.com/@oribarel/getting-the-most-out-of-your-google-colab-2b0585f82403\n",
        "\n",
        "https://research.google.com/colaboratory/local-runtimes.html"
      ]
    }
  ]
}